Subject: Re: pacing proposal

From: "Shannon Nix" <[email-masked]>
Date: Mon, 26 Jul 93 11:40:26 EST

On Thu, 22 Jul 93 17:54:34 EDT, <[email-masked]> wrote:

>
>This does not address the UI-frame balancing problem, just LLC2
>connections/circuits.  The description is informal and undoubtedly
>incomplete.  I can beef it up later if people feel the direction is
>right.
>-------------------------------------------------------------------
>
>Flags/fields in the short ("information header"):
>- Pacing Request Indicator (PREQ) - 1 bit
>- Request Larger Window Indicator (RLWI) - 1 bit
>
>New SSP message types:
>IPM
>- can be of type SOLICITED, UNSOLICITED, or RESET ACKNOWLEDGEMENT
>- can have scope of either CIRCUIT or TRANSPORT CONNECTION
>- contains Next Window Size (NWS) - 2 bytes
>
>Comments on formats -
>The NWS replaces the use of bits DWSI, IWSI, and CWI.  I don't think
>DCFI is needed in this algorithm.  I prefer that the receiver be
>able to set an explicit value for the next window size because:
>  - it gives the receiver much better control over the value
>  - it's easier to understand and work with
>

I like your adaptive pacing algorithm much better!

>The IPM fields follow the 16-byte information message header.
>
>ENTERBUSY and EXITBUSY are not part of this proposal.  Shutting off
>flow on a circuit is done with NWS=0.  Since the 6611 uses __BUSY
>signals in certain connect sequences, they may need to become part of
>the standard DLSw connect sequences, but that is another discussion.
>These signals are not to be used for normal connection flow control.
>
>
>Function:
>Pacing is done on a circuit basis only, in both directions independently.
>The pacing receiver controls via a sliding cumulative window how many
>frames a sender may put on a transport connection, for a given circuit.
>At what point a data link should be RNRed is an implementation issue
>for the router and is not specified by this pacing architecture.

I agree to this point.

>Pacing is applied only to I-frames, not UI-frames flowing on the same
>circuit.

I disagree.  UI-frames (and LLC-1 frames) should be handled through the
network in the same fashion as I-frames.  The only difference is that the
boundery DLC ( where the frame originates) is responsible for sensing 
"back-pressure" on the circuit and discarding frames as required to keep 
local buffer consuption "fair".

>
>The sender sets PREQ in the first frame of every window.  In the normal
>case, it will continue to send the full window's worth of frames before
>it must stop and await a pacing response.  If the sender reaches the
>end of a window, and has more data to send at the time the solicited
>pacing response comes back, it sets RLWI along with PREQ in the next
>pacing request.  RLWI can be set only in pacing requests.  When a pacing
>response is received, the NWS is added to the current remaining window
>size to reach a new window size.
>
>The initial pacing window size for a circuit is 2, allowing some chance
>RLWI will not be set in the next pacing request.

I would like to see the initial pacing window size for new "half-circuit"s
set in the capabilities exchange for the transport connections.

>
>In normal operation, the receiver receives I-frames and looks for
>a frame with PREQ set.  When it sees one, it determines based on
>implementation-dependent conditions the NWS to grant the sender and
>when to send the pacing response (it could be withheld).  The receiver
>has complete control over the window sizes it grants

I like it to here.

>, and is not
>required to pre-allocate or guarantee buffer space for all the outstand-
>ing windows.  

I would prefer to add:  "but is strongly recommended to" to the above 
sentence.

If the implementor does not pre-allocate his windows and is found short, 
then he has no choice but to:

1) Drop data ( not acceptable for appls built on reliable data links).
   OR
2) Gum up the transport connection.  This can cause snow-balling conditions
   throughout the net.  It also effects EVERY circuit on that connection.

I realize that a conservative statistical implementation ( using option 2 
in the case of the unlikely) for larger implementations may be acceptable.  
However, I believe your above language is a little too "laid-back".

<The receiver can get insight from the RLWI bit about how
>the sender is operating, and the receiver should honor the request if
>possible, but is not obligated to do so.
>
>Pacing responses are always sent as IPMs and are not piggy-backed.  They
>are sent at higher priority than other information flowing in the same
>direction (with TCP, as urgent data).  The responses discussed so far
>are SOLICITED and have scope CIRCUIT.

Good point ( deadlock)!

>
>To respond quickly to congestion (or reflect a received RNR), the
>pacing receiver may send an IPM of type RESET.  The NWS of a RESET IPM
>is always zero.  The pacing sender (receiver of the RESET) must
>immediately stop sending I-frames and set its current window size to
>zero.  It also sends a RESET ACK IPM at normal priority, so the
>pacing receiver can determine the end of queued data.  The RESET ACK
>is an implicit pacing request, so when the pacing receiver decides
>to reopen the window, it sends a SOLICITED IPM with a new non-zero
>window size.  All these IPMs have scope CIRCUIT.

I believe this to be unnecessary.  If sound buffer management is implemented
end-to-end, then your previously described mechanism is sufficient.  I believe
that "slamming the door shut" is

a) reactive
b) non-deterministic
c) not always appropriate

I prefer to see the receiver:

a)  Withold pacing ( possibly reducing the window size when the condition
    goes away)
OR
b)  Setting the next window size to zero.

The implication for either of these is that the "current window" is assumed
"in the pipe" ( which statistically half of it is) and must be dealt with.
That is why I favor pre-allocation of buffers as a STRONG recomendation.

To my way of thinking, a RESET IPM doesn't remove any problem, 
but rather, statistically "cuts it in half".

Further I believe the RESET IPM could have negative effects.  The 
conditions to which it is designed to react may be  "hiccups" rather than 
reasons to panic.  As long as fair, end-to-end, buffer consumption is 
enforced for the circuit, then wouldn't it be better to have data "in the 
pipeline", when the condition went way?  Reacting so quickly, would 
allow isolated, transient conditions at the boundary to create inefficient, 
bursty traffic conditions in the network.

I also think that including this feature in this proposal encourages its
misuse:  namely, using it as a determistic means of flow control.
>
>When a receiving router wishes to slam every circuit on a particular
>transport connection, it may send a RESET IPM with scope TRANSPORT
>CONNECTION.  The pacing sender must stop sending I-frames on all circuits
>on that TC and set their current window sizes to zero.  It also sends
>a RESET ACK IPM with scope TRANSPORT CONNECTION, at normal priority.
>Circuits must be restarted on an individual basis.  There is no normal
>pacing with window counts, etc., on a transport connection basis;  a
>SOLICITED IPM may never have scope TRANSPORT CONNECTION.
>

Ditto my previous comment.

>OK, folks, let me know what you think.  Maybe a consensus (like "let's
>kill this guy") will have emerged by the time I get back!

In general I think this a very good proposal!  

I thinks its advantages over the previous scheme are.

1)  Early pacing requests
2)  Unsolicited pacing responses
3)  A cleaner adaptive pacing scheme which is CONTROLLED BY THE RECEIVER!!!
4)  A partial departure from the ENTERBUSY/EXITBUSY paradigm.
5)  Elimination of "lockout" conditions by not permitting pacing responses
    to be piggy-backed

I would prefer these changes:

1)  Allowing initial window sizes to be negotiated during capabilities 
    exchange.
2)  "Stronger language" about receivers reserving buffers for 
    committed windows.
3)  Dropping RESET/RESET_ACK all together.

I would welcome debate on these points.

>Thanks,
>Peter Gayek

Great work Peter,
Shannon
(*)(*)(*)(*)(*)(*)(*)(*)(*)(*)(*)(*)(*)(*)(*)(*)(*)(*)(*)(*)(*)(*)(*)
(*)(*)(*)(*)(*)(*)(*)(*)   Shannon Nix    (*)(*)(*)(*)(*)(*)(*)(*)(*)
(*)(*)(*)(*)(*)(*)(*)      Netlink, Inc.     (*)(*)(*)(*)(*)(*)(*)(*)
(*)(*)(*)(*)(*)(*)(*)  3214 Spring Forest Rd.   (*)(*)(*)(*)(*)(*)(*)
(*)(*)(*)(*)(*)(*)(*)  Raleigh N.C. 27604 USA   (*)(*)(*)(*)(*)(*)(*) 
(*)(*)(*)(*)(*)(*)(*)  Phone:  (919)-878-8612   (*)(*)(*)(*)(*)(*)(*) 
(*)(*)(*)(*)(*)(*)(*)  Fax:    (919)-872-2132   (*)(*)(*)(*)(*)(*)(*) 
(*)(*)(*)(*)(*)(*)(*) Internet: [email-masked] (*)(*)(*)(*)(*)(*)(*) 
(*)(*)(*)(*)(*)(*)(*)(*)(*)(*)(*)(*)(*)(*)(*)(*)(*)(*)(*)(*)(*)(*)(*)

---

From: "Shannon Nix" <[email-masked]>
Date: Mon, 26 Jul 93 12:03:29 EST

On Fri, 23 Jul 93 10:58:19 EDT, Bernard Brady wrote:

>Peter,
>
>I like what you have written except in the case of the RLWI. 
>
>>In normal operation, the receiver receives I-frames and looks for a
>>frame with PREQ set.  When it sees one, it determines based on
>>has complete control over the window sizes it grants, and is not
>>required to pre-allocate or guarantee buffer space for all the
>>outstand- ing windows.  The receiver can get insight from the RLWI bit
>>about how the sender is operating, and the receiver should honor the
>>request if possible, but is not obligated to do so.
>
>I think that the receiver should be able to inform the sender that it 
>has accepted the fact of going to a larger window size.   By what I read
>the receiver is forced to accept when it sends a  pacing response.  This
>is caused by the sender automatically upping the window size when it
>receives the pacing response.   I  think we should define a Accept
>Larger Window Indicator (ALWI) - 1 bit in the pacing response.  When a
>pacing response is received with the ALWI set, the NWS is added to the
>current remaining window size to reach a new window size.  In the case
>where a RLWI is outstanding and a pacing response is received without
>the ALWI set then the window size is left unchanged by the sender. By
>witholding pacing data flow is stopped.  By immplementing ALWI the
>receiver is not accepting the NWS but is  allowing data to flow. 
>Getting data through slower is better than no data at all.
>

The way I read Peter's proposal, the receiver can set the next window size
to anything he wishes using NWS.  The RLWI is only an indication ( a 
suggestion) from the sender to the receiver that: "if you are 
able and willing to receive a bigger window next time, then I think I could 
effectively utilize it".  The receiver's acceptance or denial of that 
request is reflected in the NWS.

I will add again: I believe that the receiver should RESERVE the buffers 
for the next window before setting NWS.  That reservation may be discreet 
or statistical but NWS should reflect available buffer bandwidth.

Rgds, Shannon

>Bernie Brady